Network-aware partitioning of computation in Diamond

The Diamond storage architecture enables efficient interactive search of unindexed data by supporting the execution of application-specific filter binaries directly at storage devices. This functionality allows irrelevant objects to be discarded at the early stages of the search pipeline, thereby reducing the load on the interconnect and the user’s workstation. In order to achieve efficient use of resources under dynamic conditions, Diamond adaptively partitions computation among the storage devices and the user’s host. In this paper, we explore the behavior of Diamond systems in network-bound configurations. We develop a performance model capturing the pertinent properties of a Diamond system; in particular, we characterize the amount of network traffic generated as a result of evaluating a Diamond query. Ultimately, we formulate a partitioning algorithm that provably minimizes the amount of traffic injected into the network during the execution of a search under CPU time constraints at processing stages. Alex Nizhner ([email protected]) is with the Information Networking Institute. Larry Huston is with Intel Research Pittsburgh. Rahul Sukthankar is with Intel Research Pittsburgh and the Robotics Institute.